Papers with contextualized embeddings
Biomedical Relation Classification by single and multiple source domain adaptation (D19-62)
Copied to clipboard
| Challenge: | Existing supervised systems are highly data-driven and require a lot of effort to label data for a new domain. |
| Approach: | They propose to transfer knowledge from one or more related source domains to another domain to improve relation classification. |
| Outcome: | The proposed model outperforms neural-network based models on biomedical datasets and with contextualized embeddings on 3 biomedically-relevant datasets. |
KeywordScape: Visual Document Exploration using Contextualized Keyword Embeddings (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools for document visualization assume that keywords have static meanings, but contextualized word embeddings are unrealistic. |
| Approach: | They propose a visual exploration tool that visualizes contextualized word embeddings in documents based on keywords. |
| Outcome: | The proposed tool visualizes keywords in terms of their contextualized embeddings in a semantic landscape that keeps keywords with similar context close to each other, allowing for a more precise search and comparison of documents. |
Synthetic Propaganda Embeddings To Train A Linear Projection (D19-50)
Copied to clipboard
| Challenge: | Using contextualized token embeddings, we can extract features of propaganda from contextualized embeddnings without fine-tuning the large parameters of the base model. |
| Approach: | They propose a method for detecting fine-grained categories of propaganda in text by generating synthetically generated embeddings from pre-trained language models. |
| Outcome: | The proposed method is used in the first shared task in fine-grained propaganda detection at NLP4IF as Team Stalin. |
Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction (D19-1)
Copied to clipboard
| Challenge: | Existing systems treat this task as a pipeline of two separate subtasks, i.e., event extraction and temporal relation classification. |
| Approach: | They propose a joint event and temporal relation extraction model with shared representation learning and structured prediction. |
| Outcome: | The proposed method improves both event extraction and temporal relation extraction over state-of-the-art systems. |
Word class flexibility: A deep contextualized approach (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on word class flexibility have been fraught with difficulties in quantifying it accurately and at scale. |
| Approach: | They propose a method to quantify word class flexibility in 37 languages using contextualized word embeddings. |
| Outcome: | The proposed method builds on recent work in contextualized word embeddings to quantify semantic shift between word classes and uncovers shared tendencies in class flexibility across languages. |
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for document summarization require human annotations and annotations. |
| Approach: | They propose a method which measures the quality of a summary by measuring its semantic similarity with a pseudo reference summary, using contextualized embeddings and soft token alignment techniques. |
| Outcome: | The proposed method correlates better with human ratings by 18- 39% compared to the state-of-the-art evaluation metrics. |
Semantic Frame Induction with Deep Metric Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown the usefulness of contextualized word embeddings in semantic frame induction, but they are not always consistent with human intuitions about semantic frames. |
| Approach: | They propose a model that fine-tunes contextualized embeddings to perform semantic frame induction. |
| Outcome: | The proposed model improves clustering evaluation scores on FrameNet by 8 points or more. |
SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word alignments are useful for statistical and neural machine translation (NMT) and cross-lingual annotation projection. |
| Approach: | They propose to leverage multilingual word embeddings for word alignment. |
| Outcome: | The proposed methods perform better for four languages and comparable for two languages than traditional statistical aligners even with abundant parallel data. |
A Few Topical Tweets are Enough for Effective User Stance Detection (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work that employs unsupervised classification has shown that user stance detection is highly accurate on vocal Twitter users, but fails for less vocal users, who may have only authored a few tweets about a target. |
| Approach: | They propose to perform user stance detection on vocal Twitter users using contextualized embeddings to represent tweets and then perform unsupervised classification of the user. |
| Outcome: | The proposed method outperforms two baseline methods and achieves 89.6% accuracy and 91.3% macro F-measure on eight controversial topics. |
The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new dataset evaluates the credibility of statements made by Israeli public figures and politicians . a dataset of 1021 statements is used to assess the credibility and accuracy of statements . |
| Approach: | They propose a dataset to evaluate the credibility of statements by Israeli politicians . they use annotated statements manually annotating them for their credibility status . |
| Outcome: | The proposed model outperforms models based on statement and context, and achieves a 48.3 F1 score. |
Computational Modeling of Affixoid Behavior in Chinese Morphology (2020.coling-main)
Copied to clipboard
| Challenge: | affixoid behavior in Mandarin Chinese is unclear due to polysemy and diachronic dynamics. |
| Approach: | They propose to use three quantitative features to model affixoid behavior in Mandarin Chinese to determine its status. |
| Outcome: | The proposed model shows that there are no clear criteria that can be used to identify an affix’s status in an isolating language like Mandarin Chinese. |
Effect of Post-processing on Contextualized Word Representations (2022.coling-1)
Copied to clipboard
| Challenge: | Post-processing of static embeddings has been shown to improve their performance on both lexical and sequence-level tasks. |
| Approach: | They standardize individual neuron activations using z-score, min-max normalization, and remove top principal components using the all-but-the-top method. |
| Outcome: | The proposed method unwraps vital information present in the representations for both lexical and sequence classification tasks. |
Do Transformer Networks Improve the Discovery of Rules from Text? (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing rules that model binary relations such as X treat Y are based on the distributional hypothesis of Harris (1954). |
| Approach: | They propose to implement the distributional hypothesis using contextualized embeddings provided by a transformer-network-based language model to measure the similarity between slots instead of lexical overlap. |
| Outcome: | The proposed approach outperforms the original DIRT algorithm in the question answering-based evaluation. |
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings are becoming a ubiquitous component of natural language processing. |
| Approach: | They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains . |
| Outcome: | The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter. |
RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English) (2021.acl-long)
Copied to clipboard
| Challenge: | lexical ambiguity is a problem for NLP, but few tasks evaluate its impact on human intuitions. |
| Approach: | They propose to use contextualized word embeddings to evaluate word meaning . they use a dataset of human relatedness judgments and human estimates of sense dominance . |
| Outcome: | The proposed model matches human intuitions with contextualized embeddings on 112 ambiguous words in context with 672 sentence pairs. |
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual word embeddings in a contextual space are lacking. |
| Approach: | They propose a method to generate cross-lingual contextualized word embeddings using pre-trained BERT models by learning a linear transformation from contextual word alignments. |
| Outcome: | The proposed approach outperforms state-of-the-art models on zero-shot cross-lingual transfer parsing and is highly competitive with existing models. |
SAMRank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT-2 (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for keyphrase extraction use contextualized embeddings to capture semantic relevance between words, sentences, and documents. |
| Approach: | They propose an unsupervised keyphrase extraction approach that uses only a self-attention map in a pre-trained language model to determine the importance of phrases. |
| Outcome: | The proposed approach outperforms embedding-based models on three keyphrase extraction datasets. |
Clinical Concept Linking with Contextualized Neural Representations (2020.acl-main)
Copied to clipboard
| Challenge: | Entity linking systems rely on three sources of information: 1) similarity between mention string and entity name; 2) similarity of context of document to entity; 3) broader information about knowledge base; 4) contextual information; 5) semantic information; and 6) semantic information. |
| Approach: | They propose an approach to linking medical concepts to a medical concept ontology that leverages recent work in contextualized neural models. |
| Outcome: | The proposed approach outperforms a baseline approach and provides better initialization for the ranker. |
Automatic Correction of Syntactic Dependency Annotation Differences (2022.lrec-1)
Copied to clipboard
| Challenge: | Annotation inconsistencies between data sets can cause problems for low-resource NLP . a simple method for automatically detecting annotation mismatches between corpora is proposed . |
| Approach: | They propose a method for automatically detecting annotation mismatches between dependency parsing corpora . they also propose three related methods for automatically configuring the mismatch . |
| Outcome: | The proposed method improves performance on both converted and unconverted data. |
PMI-Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that using contextualized embeddings from pre-trained multilingual language models could give us high quality word alignments without the need of parallel training data. |
| Approach: | They propose a method which uses contextualized embeddings from pre-trained language models to extract word alignments without parallel training. |
| Outcome: | The proposed method outperforms rival methods on five out of six language pairs. |
Knowledge-augmented Graph Neural Networks with Concept-aware Attention for Adverse Drug Event Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have used word embedding and deep learning to automate ADE detection from text, but they did not incorporate explicit medical knowledge about drugs and adverse reactions or the corresponding feature learning. |
| Approach: | They propose to integrate medical knowledge into ADE detection from text . they use contextualized embeddings from pretrained language models and convolutional graph neural networks to learn features differently for different types of nodes in the graph. |
| Outcome: | The proposed model outperforms existing models on four public datasets and shows that it is based on medical knowledge and embeddings from pretrained language models and neural networks. |
Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a cognitive science research focus on aligning language spaces in their entirety . but, cognitive science has long focused on a local perspective . a new method for cross-lingual lexical alignment requires some methodology . |
| Approach: | They propose a method for analyzing kinship domain kinematics and a new method for contextualization . they propose kin-level validations and contextualizations to validate the results . |
| Outcome: | The proposed method analyzes synthetic validations and naturalistic validations using lexical gaps in the kinship domain. |
Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that transformer language models encode psychologically meaningful aspects of meaning at different depths. |
| Approach: | They conduct a layer-wise probing study of 58 psycholinguistic features across 10 transformer models . they find that apparent localization of meaning is method-dependent . |
| Outcome: | The results show that where meaning “lives” in transformer models reflects methodological choices and architectural constraints. |